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1  INTRODUCTION 


In  this  project  we  acquired  and  registered  a  multimodal  dynamic  target  image  test  set  that 
can  be  used  to  assess  the  operational  effectiveness  of  static  and  dynamic  image  fusion 
techniques  for  a  range  of  relevant  military  tasks. 

The  fusion  of  multimodality  (multi-spectral)  images  and  video  sources  is  emerging  as  a 
vital  technique  for  surveillance  purposes,  navigation  and  object  tracking  applications.  The 
main  goal  of  image  fusion  is  to  provide  a  single  compact  representation  of  the  input 
images  that  is  more  informative  than  each  of  the  individual  inputs.  There  are  several 
potential  benefits  of  multi-sensor  image  fusion:  wider  spectral,  spatial  and  temporal 
coverage,  extended  range  of  operation,  decreased  uncertainty,  improved  reliability  and 
increased  robustness  of  the  system  performance. 

In  many  applications  the  human  perception  of  the  fused  image  is  of  fundamental 
importance.  As  a  result,  image  fusion  results  are  mostly  evaluated  by  human  visual 
inspection.  Subjective  evaluation  tests  are  often  time-consuming,  expensive,  impractical 
or  even  outright  dangerous.  Moreover,  such  methods  often  do  not  correlate  well  with  how 
people  perform  in  actual  tasks  utilizing  fused  images.  This  has  led  to  an  increasing 
demand  for  efficient  and  objective  tests  that  allow  rapid  comparison  of  the  results 
obtained  with  different  algorithms,  or  the  automatic  selection  of  the  appropriate  fusion 
algorithm  for  a  given  scenario,  or  to  obtain  the  optimal  settings  for  a  specific  fusion 
algorithm. 

Effective  image  fusion  systems  should  provide  a  more  complete  representation  (with 
increased  information  content)  of  the  scene,  which  is  easier  to  interpret  and  understand 
(ergonomic  value).  A  range  of  different  image  fusion  algorithms  is  currently  available, 
many  of  which  can  be  implemented  in  real-time.  In  practice  many  image  fusion 
algorithms  merely  produce  fused  images  with  an  increased  amount  of  detail  (compared  to 
the  original  input  images)  without  taking  into  account  the  information  content  (the 
meaning)  of  the  resulting  combined  details.  As  a  result,  the  perceptibility  of  relevant 
features  in  the  fused  representation  of  the  scene  may  be  degraded,  and  task  performance 
may  be  adversely  affected.  For  instance,  when  clutter  is  included  in  the  fusion  process  a 
large  number  of  spurious  (i.e.  non-informative  or  task  irrelevant)  details  may  appear  in 
the  resulting  fused  image.  As  a  result  human  or  machine  performance  of  target 
detectability  and  target  classification  may  be  severely  degraded. 

Image  quality  is  task  related.  A  fused  image  can  be  said  to  be  of  good  quality  if  it  allows 
the  observer  to  achieve  a  task  performance  that  is  similar  to  or  better  than  the 
performance  that  can  be  achieved  with  the  original,  individual  images.  However,  the 
quantification  of  this  performance  assessment  is  often  difficult  and  time  consuming. 
Hence,  there  is  a  need  for  efficient  and  reliable  methods  to  quantify  the  operational 
effectiveness  of  image  fusion  systems. 
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The  multimodal  nightvision  imagery  collected  in  this  study  can  and  will  be  used  to  (1) 
evaluate  existing  image  fusion  schemes,  (2)  to  design  and  optimize  new  dynamic  image 
fusion  schemes,  and  (3)  to  develop  new  image  fusion  quality  metrics.  In  the  next  sections 
we  will  give  a  detailed  account  of  the  equipment,  the  scenario,  the  targets,  the  registration 
location,  and  the  environmental  conditions. 
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2  EQUIPMENT 


An  instrumented  truck  (see  Figure  lb)  was  used  to  house  the  equipment  (computers  and 
monitors)  that  was  used  to  register  and  monitor  the  imagery. 

The  sensor  suite  (see  Figure  lb)  was  placed  just  outside  the  truck  and  consisted  of 

-  A  Lion  Advance  8-12  micrometer  longwave  thermal  camera  (Thales  Optronics), 
with  a  field-of-view  of  7. 8x5. 9  degrees,  a  focal  point  distance  of  81.71  mm,  a 
detector  pitch  of  35  p,  and  a  NETD<80mK. 

-  A  Raytheon  Radiance  High  Speed,  256x256  pixels,  InSb  focal  plane  array 
midwave  thermal  camera  (Raytheon). 

-  A  digital  image  intensifier  (DEP),  with  a  field-of-view  of  8. 1x6.1  degrees. 

The  analog  signals  from  all  cameras  were  digitized  using  a  Matrox  MIL  Lite 

framegrabber. 


GPS  signals  were  continuously  registered  both  at  the  location  of  the  sensor  suite  and  at 
the  location  of  the  targets.  During  the  experiment  the  soldiers  carried  a  backpack  with  a 
laptop  (Dell  Inspirion)  that  was  attached  to  a  BU-353  USB  SiRF  Star  III  GPS  receiver. 
An  identical  combination  of  GPS  receiver  and  laptop  was  placed  next  to  the  sensor  suite. 
The  difference  between  these  two  GPS  signals  at  a  given  time  corresponds  to  the  distance 
between  the  target  and  the  sensor  suite  at  that  time. 
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Figure  1.  Sensor  suite  (a)  situated  just  outside  the  registration  truck  and  (b)  in  close-up.  Left:  the  Lion 
Advance  8-12  pm  thermal  camera;  upper  middle:  the  digital  image  intensifier;  right:  the  Radiance  HS  3-5 
pm  midwave  camera. 
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3  SCENARIO 


The  scenario  was  as  follows.  A  soldier,  wearing  standard  camouflage  clothing  and 
carrying  a  target  object,  approached  the  sensor  suite  along  a  straight  line,  starting  at  a 
distance  of  approximately  300  m.  When  the  soldier  had  neared  the  cameras  to  a  distance 
of  about  40  m  the  image  collection  stopped  and  he  retuned  along  the  same  route  to  his 
starting  position.  Each  time  the  soldier  reached  the  starting  point  he  exchanged  the  target 
object  for  another  one.  Then  the  image  registration  started  again  and  the  soldier  walked  at 
a  steady  pace  towards  the  cameras.  This  procedure  was  repeated  until  all  targets  had  been 
used  once.  All  image  registrations  were  done  after  dark.  In  these  conditions,  the  targets 
were  not  visible  with  the  naked  eye  for  the  largest  part  of  the  track  (they  could  only  be 
noticed  at  close  range). 


(c) 


Figure  2.  Corresponding  images  from  an  approach  sequence,  taken  respectively  with  (a)  the  Radiance  HS 
3-5  pm  midwave  camera,  (b)  the  Lion  Advance  8-12  pm  thermal  camera,  and  (c)  the  digital  image 
intensifier.  The  images  show  a  soldier  carrying  a  target  object.  The  recordings  were  made  in  complete 
darkness,  such  that  the  target  was  not  visible  with  the  naked  eye. 
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Figure  3.  GPS  tracks  of  soldiers  approaching  the  cameras  (lower  direction)  from  afar  (upper  position)  and 
turning  around  to  their  starting  position. 
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4  TARGETS 


Figure  4  shows  images  of  all  the  targets  that  were  used  in  this  experiment.  The  targets 
were  respectively  a  wooden  hammer,  a  wooden  stake,  an  axe,  a  Mach  machine  gun,  an 
Ml 6,  a  Minime  and  a  Glock  handgun.  Table  1  lists  the  characteristic  dimensions  (length 
and  width)  of  all  these  targets. 

Table  1.  targets  and  their  characteristic  dimensions. 


Target 

Length 

(cm) 

Width  (cm) 

Hammer 

40 

19 

Stake 

81 

5 

Axe 

88 

22 

Mini  SAW 

91 

- 

M-16 

40 

26 

M-60  light  machine  gun 

126 

- 

Glock 

18 

13 
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Figure  4.  The  targets:  (a)  hammer,  (b)  stake,  (c)  axe,  (d)  mini  SAW,  (e)  M-16,  (f)  M-16  with  grenade 
launcher,  (g)  M-60  light  machine  gun,  (h)  Glock. 
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5  REGISTRATION  LOCATION 


The  Vlasakkers  registration  site  was  located  on  a  military  base  in  Amersfoort,  The 
Netherlands.  Figure  5  shows  the  registration  location  in  daytime.  The  background 
consisted  of  a  forest  line.  The  terrain  to  the  left  of  the  track  consisted  of  heather  and 
grass,  which  provided  a  cluttered  background.  To  the  right  of  the  track  there  was  a  tarmac 
road.  The  soldiers  walked  towards  the  camera  standpoint  from  a  distance  of  about  300  m 
(near  the  treeline  in  the  back).  Their  path  was  near  the  edge  of  the  road,  just  along  the 
grass. 


Figure  5.  Registration  location  in  daytime.  During  image  registration  at  night  soldiers  approached  the 
caneras  from  the  distance,  walking  along  the  edge  of  the  road  near  the  grass. 
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Figure  6.  The  GPS  track  data  projected  onto  a  Google  Earth  satellite  view  of  the  registration  location. 
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6  ENVIRONMENTAL  CONDITIONS 


The  image  collection  was  performed  during  the  night  of  Tuesday  9  September  2008.  The 
weather  was  quite  variable  during  the  registration  period. 

At  the  start  of  the  registration  period  it  was  dry  and  the  visibility  was  good.  After 
completing  5  consecutive  runs  it  started  to  drizzle.  The  next  1 1  runs  were  performed 
during  the  rain.  At  the  end  of  the  experiment  it  was  dry  again.  Table  2  lists  the 
meteorological  data  at  the  time  and  location  of  the  field  trial. 

Table  2.  Weather  data  of  Tuesday  9  September  2008  at  de  Bilt,  near  the  registration  site.  Source:  The 
Royal  Dutch  Meteorological  Institute. 


Weather  data  of  tuesday  9  September  2003  at  De  Bilt 

Temperature 

Average 

Precipitation 

Mean 

17.3  °C 

15.2  °C 

24h  sum 

2.9  mm 

Maximum 

23.3  °C 

20.0  DC 

Duration 

1.5  hours 

Minimum 

10.0  °C 

10.4  °C 

Sun,  cloud  cover  Sl  visibility 

Wind 

Duration  sunshine 

10.1  hours 

Mean 

3.0  m/s 

=  2  Eft 

Relative  sunshine  duration 

77  % 

37  % 

Maximum  hourly  mean 

5.0  m/s 

=  3  Eft: 

Average  cloud  cover 

4  o eta's 

Maximum  gust 

11.0  m/s 

partly  cloudy 

Minimum  visibility 

0.4  km 

Prevailing  direction 

151  D 

=  SEE 

Relative  atmospheric  humidity 

Air  pressure 

Mean 

79  % 

32  % 

Mean  air  pressure 

1014.1  hPa 
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CONCLUDING  REMARKS 


At  total  of  16  runs  were  successfully  registered,  partly  in  dry  and  partly  in  rainy 
conditions. 

The  imagery  collected  during  this  study,  together  with  the  GPS  data  and  all  other 
documentary  material,  was  made  available  to  the  Air  Force  Research  Laboratory 
organization  at  the  end  of  the  field  trials. 

Researchers  from  both  the  Air  Force  Research  Laboratory  and  TNO  Human  Factors  will 
write  a  joint  paper  on  (a)  laboratory  experiments  that  will  be  performed  in  both  labs  with 
the  imagery  that  has  been  collected  in  the  present  study,  and  (b)  on  the  way  in  which  this 
can  be  further  deployed  for  future  laboratory  experiments.  This  joint  paper  will  be 
presented  at  the  SPIE  Defense,  Security  +  Sensing  Symposium  in  Orlando,  Florida,  13-17 
April  2009. 
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